Dynamic range estimation with fast and slow sensor pixels

ABSTRACT

A scene can be captured by integrating a first sensor pixel for a first amount of time to produce an original first photometric and integrating a second sensor pixel for the first amount of time to produce an original second photometric. The first sensor pixel can be configured to saturate with photocharge slower than the second sensor pixel. The scene can be recaptured by integrating the second sensor pixel for a second amount of time less than the first amount of time.

BACKGROUND Field of the Disclosure

The present application relates to digital imaging.

Description of Related Art

Scene dynamic range can describe the range of light intensities in ascene. A scene including the sun and dark shadows will have a widedynamic range. A scene of a dark, foggy day will have a narrow dynamicrange. Similarly, a scene of a bright sky will have a narrow dynamicrange.

Light intensity and therefore, dynamic range, can be expressed in termsof unfiltered light or filtered light. A color filter can admit desiredlight while blocking undesired light. For example, a green filter admitslight falling in the green spectrum while blocking light falling outsidethe green spectrum. As a result, green scene dynamic range can be foundby passing scene light through a green color filter.

Digital cameras often include automatic exposure control. Duringautomatic exposure control, a digital camera can measure (e.g.,approximate) scene dynamic range, then adjust exposure to match thescene dynamic range. Without automatic exposure control, a digitalcamera can produce an underexposed image or an overexposed image.

For an underexposed image, some of the digital camera's sensor pixelswill receive an underdose of light, causing the sensor pixels to recorderroneously small channel values (i.e., undersaturate). For example, afirst group of sensor pixels may capture ocean and a second group ofsensor pixels may capture beach. During underexposure, both the firstand second groups can fail to activate (e.g., capture photometrics ofzero). The resulting image would not distinguish between ocean andbeach.

For an overexposed image, some of the digital camera's sensor pixelswill receive an overdose of light, causing the sensor pixels to recorderroneously large channel values (i.e., oversaturate). For example, afirst group of sensor pixels may capture sun and a second group ofsensor pixels may capture bright sky. During overexposure, both thefirst and second groups can fully activate (e.g., deliver maximumphotometrics). The resulting image would not properly distinguishbetween sky and sun.

SUMMARY

A method can include capturing a scene by integrating a first sensorpixel for a first amount of time to produce an original firstphotometric and integrating a second sensor pixel for the first amountof time to produce an original second photometric. The first sensorpixel can be configured to saturate with photocharge slower than thesecond sensor pixel.

The method can include re-capturing the scene by integrating the secondsensor pixel for a second amount of time to produce a newer secondphotometric. The second amount of time can be based on the originalfirst photometric. The second amount of time can be less than the firstamount of time.

A processing system can include one or more processors configured toperform the method. The method can be stored as program code in anon-transitory computer-readable medium. When executed, the code cancause one or more processors to perform the method.

A method can include capturing a scene with a scene dynamic range byintegrating a plurality of first sensor pixels for a first amount oftime to produce a plurality of original first photometrics andintegrating a plurality of second sensor pixels for the first amount oftime to produce a plurality of original second photometrics. Each of theplurality of first sensor pixels can be configured to saturate withphotocharge slower than each of the plurality of second sensor pixels.

The method can include normalizing the plurality of original firstphotometrics with respect to the plurality of second sensor pixels. Themethod can include determining whether at least some of the plurality ofsecond sensor pixels clipped a top end of the scene dynamic range basedon the normalized plurality of original first photometrics.

The method can include re-capturing the scene by integrating theplurality of second sensor pixels for a second amount of time to producea plurality of newer second photometrics based on determining that atleast some of the plurality of second sensor pixels clipped the top endof the scene dynamic range. The second amount of time can be less thanthe first amount of time.

A processing system can include one or more processors configured toperform the method. The method can be stored as program code in anon-transitory computer-readable medium. When executed, the code cancause one or more processors to perform the method.

A non-transitory computer-readable medium can include program code,which, when executed by one or more processors, causes the one or moreprocessors to perform operations. The program code can include code for:capturing a scene by integrating a first sensor pixel for a first amountof time to produce an original first photometric and integrating asecond sensor pixel for the first amount of time to produce an originalsecond photometric. The first sensor pixel can be configured to saturatewith photocharge slower than the second sensor pixel.

The program code can include code for: re-capturing the scene byintegrating the second sensor pixel for a second amount of time toproduce a newer second photometric. The second amount of time can bebased on the original first photometric. The second amount of time canbe less than the first amount of time.

A processing system can include means for capturing a scene byintegrating a first sensor pixel for a first amount of time to producean original first photometric and integrating a second sensor pixel forthe first amount of time to produce an original second photometric. Theprocessing system can include means for re-capturing the scene byintegrating the second sensor pixel for a second amount of time toproduce a newer second photometric. The second amount of time can bebased on the original first photometric. The second amount of time canbe less than the first amount of time.

BRIEF DESCRIPTION OF DRAWINGS

The above summary and the below detailed description of illustrativeembodiments may be better understood when read in conjunction with theappended Figures. The Figures show some of the illustrative embodimentsdiscussed herein. For clarity and ease of reading, some Figures omitviews of certain features. Unless expressly stated otherwise, theFigures are not to scale and features are shown schematically.

FIG. 1 shows example mobile devices imaging a scene.

FIG. 2 shows a rear view of an example mobile device.

FIG. 3 shows a cross-sectional plan view of an example camera.

FIG. 4 is a block diagram of an example processing system.

FIG. 5-7 show example sensor pixels in example sensor panels.

FIGS. 8-14 show example sensor pixel configurations.

FIG. 15 shows a method of performing automatic exposure control based onslow and fast photodiodes (i.e., slow and fast sensor pixels).

FIG. 16 shows a method of performing automatic exposure control based ontwo or more of slow, fast, and fastest photodiodes (i.e., slow, fast,and fastest sensor pixels).

FIG. 17 shows histograms of an example use case consistent with themethods of FIGS. 15 and 16

FIG. 18 is a block diagram showing relationships between photodiodes.

DETAILED DESCRIPTION

The present application includes illustrative embodiments. Some claimedimplementations will have different features than in the illustrativeembodiments. Changes can be made without departing from the spirit ofthe disclosure. For example, features of the illustrative embodimentscan be incorporated in different systems (e.g., devices) and methods.

At times, the present application uses directional terms (e.g., front,back, top, bottom, left, right, etc.) to give the reader context whenviewing the Figures. Directional terms are for the reader's convenienceand do not limit the claimed inventions to a particular orientation. Anyabsolute term can be replaced with a relative term (e.g., fast can bereplaced with faster, slow can be replaced with slower). Any relativeterm can be replaced with a numbered term (e.g., left can be replacedwith first, right can be replaced with second, and so on).

Among other things, the present application discloses techniques fordetermining (e.g., estimating, approximating, projecting) a scenedynamic range of a scene captured by a camera. A camera can include asensor panel with first (e.g., slower) sensor pixels and second (e.g.,faster) sensor pixels. All else being equal, first sensor pixels cantake longer to saturate with photocurrent than second sensor pixels.First sensor pixels can be used to approximate a ceiling of the scenedynamic range while second sensor pixels can be used to approximate afloor of the scene dynamic range, as discussed in further detail below.

The processing system can perform automatic exposure control based onthe scene dynamic range. During automatic exposure control, theprocessing system can adjust internal parameters of a camera to spreadout sensor pixel measurements and to prevent clipping.

A top (e.g., ceiling) of a scene dynamic range is clipped when a camerarecords (i.e., captures) the same maximum photometrics for scene objectsA and B. This is called overexposure. A bottom (e.g., floor) of a scenedynamic range is clipped when a capture records the same minimumphotometrics for scene objects A and B. This is called underexposure.

The processing system can improve the image by spreading or divergingthe channel values through automatic exposure control until the camerarecords sufficiently distinct photometrics for scene object A and sceneobject B. To correct overexposure, the processing system can increasethe maximum scene light intensity that sensor pixels can capture. As aresult, the camera can lift the dynamic range ceiling of the sensorpixels. To correct underexposure, the processing system can reduce theminimum light intensity that sensor pixels can capture. As a result, thecamera can lower the dynamic range floor of the sensor pixels. Tocapture an entire span of scene dynamic range, the sensor pixel dynamicrange floor can be less than or equal to the scene dynamic range floorand the sensor pixel dynamic range ceiling can be greater than or equalto the scene dynamic range ceiling.

Various techniques to increase exposure (i.e., fix underexposure) andreduce exposure (i.e., fix overexposure) are known in the art. Toexecute automatic exposure control, the processing system can performone or more of the following example techniques.

First, the processing system can adjust sensor pixel integration time(i.e., how long each sensor pixel is effectively exposed to light).Integration time positively correlates with exposure level. Therefore,integration time can be increased during an underexposure event andreduced during an overexposure event.

Second, the processing system can adjust sensor pixel gain. Theadjustment can be non-linear. To correct underexposure, the gain for lowsensor pixel measurements can be nonlinearly increased. To correctoverexposure, the gain for high sensor pixel measurements can benonlinearly reduced.

Third, the processing system can adjust aperture size. Aperture sizepositively correlates with exposure level. To correct an underexposureevent, aperture size can be increased. To correct an overexposure event,aperture size can be reduced.

FIG. 1 shows mobile devices 100 imaging a scene 10 a. Mobile device 100can be a smartphone 100 a or a dedicated camera assembly 100 b. FIG. 2shows a rear view of smartphone 100 a. Mobile device 100 can include aprocessing system 400 (schematically shown in FIG. 4 and discussedbelow). Processing system 400 (PS 400) can include one or more cameras101 (also called optical sensors, image sensors, and the like). Camera101 can be a sensor 404 of PS 400 and include one or more of: one ormore processors 401 (e.g., circuitry of sensor panel 121), memory 402,input-output devices 403, sensors 404 (e.g., photodiodes 805 of sensorpanel 121), user-interfaces 405, and motors/actuators 406 (e.g.,auto-focus motors). PS 400 can thus represent components that can bewithin camera 101 and/or a more global system that can include camera101. Alternatively, camera 101 can be a component separate fromprocessing system 400.

In some embodiments, PS 400 can be a processing system of mobile device100 (e.g., one or more of smartphone 100 a or dedicated camera assembly100 b). Besides camera 101, mobile device 100 can include, for example,a frame (not labeled), a display 102, and hard buttons 103. PS 400 canbe configured to present soft or virtual buttons 104 on display 102.

As used herein, camera 101 can be any type of one or more digital imagesensors. Therefore, the term “camera” encompasses digital image sensorsof dedicated camera assemblies 100 b and digital image sensors mountedto any type of device (e.g., a smartphone 100 a, a vehicle, etc.).

PS 400 can be configured to enter a viewfinder mode. During viewfindermode, PS 400 can present a stream of full-color images (e.g., samples offull-color images) on display 102. When the user presses a capturebutton (e.g., buttons 103 or 104), PS 400 can be configured to preservea stable image in memory.

PS 400 can preserve the stable image in memory indefinitely (e.g., innon-volatile memory). The capture button can cause mobile device 100 torecord a single image or multiple images, which represent frames of avideo. Stable images are further discussed below.

FIG. 3 is a schematic view of camera 101 (also called an image sensorpackage), which can be mounted in mobile device 100, or any other kindof system (e.g., a vehicle). Camera 101 can be representative of some orall of cameras 101 a-101 e shown in FIGS. 1 and 2.

Camera 101 can include a housing 111 retaining a lens 112 and a sensorpanel 121 (also called an image sensor). As shown in FIG. 3, lens 112can admit light 301 from a scene (e.g., scene 10 a), and outputconverged light 301. A first portion of converged light 301 can bein-focus (e.g., substantially in-focus) on sensor panel 121. Therefore,the first portion of light 301 can represent the field of focus. Theremainder of converged light 301 can be out-of-focus on sensor panel121. Sensor panel 121 can measure converged light 301 with sensor pixels501 (discussed below).

Although not shown, camera 101 can include multiple lenses and otheroptical elements such as moveable apertures, shutters, mirrors, and thelike. Camera 101 can lack a moveable aperture (e.g., only include afixed aperture). Camera 101 can include an actuator configured to moveat least one lens. The actuator can be a motor. The actuator can movethe lens in response to an autofocus command. PS 400 can issue anautofocus command based on phase detection sensor pixels (i.e., performphase detection autofocus), as is known in the art.

Referring to FIG. 4, and as further addressed below, PS 400 can includeone or more processors 401 and memory 402. According to some examples,camera(s) 101, display 102, and hard buttons 103, are aspects of PS 400(e.g., depicted as sensors 404, UI(s) 405, and/or I/O 403). PS 400 canbe configured to perform any of (e.g., all of) the functions,operations, and methods discussed in the present application. PS 400 canbe present in any kind of device (e.g., a vehicle, a desktop computer, asmartphone, etc.).

FIGS. 5-7 show example configurations of sensor panel 121 from aschematic top plan perspective. Sensor panel 121 can include differentkinds of sensor pixels 501. “G” sensor pixels 501 g can include a greencolor filter. “B” sensor pixels 501 b can include a blue color filter.“R” sensor pixels 501 r can include a red color filter. Any instance ofthe term “G” in the present application can be replaced with, forexample, “first color” or “first spectral channel”. Any instance of theterm “B” in the present application can be replaced with, for example,“second color” or “second spectral channel”. Any instance of the term“R” in the present application can be replaced with, for example, “thirdcolor” or “third spectral channel”.

Sensor panel 121 can include phase detection (“PD”) sensor pixels 501pd. PD sensor pixels 501 pd can include any kind of color filter such asgreen, blue, red, infrared, clear, etc. In some embodiments, PD sensorpixels 501 pd can lack a color filter.

Sensor panel 121 can include sensor pixels 501 with other types of colorfilters, such as clear color filters, infrared filters, etc. Some sensorpixels 501 can lack a color filter. Example sensor pixels 501 arediscussed below with reference to FIGS. 8-13.

Although sensor pixels 501 are shown as being consecutive, sensor panel121 can define gaps between adjacent sensor pixels 501 (not shown). Thegaps (not shown) can be used for read-out circuitry, etc. FIGS. 5-7schematically show sensor panel 121 as including sixty-four sensorpixels 501. In various embodiments, sensor panel 121 can includethousands or millions of sensor pixels 501.

Sensor pixels 501 can be categorized as slow sensor pixels 502 or fastsensor pixels 503. Example features of slow sensor pixels 502 and fastsensor pixels 503 are further discussed below. Each of the sensor pixels501 can include photodiodes 805. Slow sensor pixels 502 can include slowphotodiodes 806. Fast sensor pixels 503 can include fast photodiodes807.

The terms fast and slow are relative, not absolute. For example, fastsensor pixels 503 can be faster than slow sensor pixels 502, but slowerthan fastest (i.e., faster) sensor pixels 503. As another example, andreferring to the example embodiment in FIG. 13 (further discussedbelow), photodiodes 1201, 1203, and 1204 are fast compared withphotodiode 1202 and photodiodes 1202, 1203, and 1204 are slow compareswith photodiode 1201.

Slow photodiodes 806 can be configured to saturate with photochargeslower than fast photodiodes 807. For example, and all else being equal,slow photodiodes 806 can take “X” amount of time to saturate withphotocharge and fast photodiodes 807 can take “X”/2 amount of time tosaturate with photocharge. In some embodiments, slow sensor pixels 502can be shielded 811 while fast sensor pixels 503 can be unshielded. Slowsensor pixels 502 can include more shielding than fast sensor pixels503.

Referring to FIG. 5, and in some embodiments, PD sensor pixels 501 pdcan be slow sensor pixels 502 while the remaining sensor pixels 501r,g,b can be fast sensor pixels 503. In other embodiments, PD sensorpixels 501 pd can be slow sensor pixels 502 while at least some of theremaining sensor pixels 501 r,g,b can be slow sensor pixels 502 and atleast some of the remaining sensor pixels 501 r,g,b can be fast sensorpixels 503.

Similarly, in FIG. 6, and according to some embodiments, PD sensorpixels 501 pd can be slow sensor pixels 502 while the remaining sensorpixels 501 r,g,b can be fast sensor pixels 503. According to otherembodiments, PD sensor pixels 501 pd can be fast sensor pixels 503 whilethe other sensor pixels 501 r,g,b are slow sensor pixels 502. In furtherembodiments, PD sensor pixels 501 pd can be slow sensor pixels 502 whilesome of the other sensor pixels 501 r,g,b are slow sensor pixels 502 andsome of the other sensor pixels 501 r,g,b are fast sensor pixels 503.

In FIG. 7, and according to various embodiments, any (e.g., all) of theillustrated R, G, B sensor pixels 501 r,g,b can function as PD sensorpixels 501 pd. Any of the sensor pixels 501 of FIG. 7 can be slow sensorpixels 502 or fast sensor pixels 503. As addressed below, each of thesensor pixels 501 of FIG. 7 can include fast photodiodes 807 and slowphotodiodes 806.

Referring to FIGS. 5-7, and in some embodiments, a predeterminedfraction (e.g., 1/10) of G, B, and R sensor pixels 501 g,b,r can be slowsensor pixels 502 while the remaining fraction (e.g., 9/10) of G, B, andR sensor pixels 501 r,g,b can be fast sensor pixels 503. In otherembodiments, only one category of G, B, and R sensor pixels 501 r,g,bcan include slow sensor pixels 502. For example, a predeterminedfraction (e.g., 1/10) of G sensor pixels 501 g can be slow sensor pixels502 while the remaining fraction (e.g., 9/10) of G sensor pixels 501 gcan be fast sensor pixels 503. According to this example, all B and Rsensor pixels 501 b,r can be fast sensor pixels 503.

FIGS. 5-7 show only three of many different possible sensor panel 121configurations. Therefore, sensor panels 121 of FIGS. 5-7, as with allfeatures disclosed in the present application, are only examples. Sensorpanel 121 can have many different configurations.

FIGS. 8-14 show possible sensor pixels 501. FIGS. 8-11 are in crosssectional side elevational schematic. In FIGS. 8-11, light flows fromtop to bottom. FIGS. 12-14 are in top plan schematic. In FIGS. 12-14,light flows into the page. The sensor pixels 501 shown in FIGS. 8-14 canbe representative of any of the sensor pixels 501 in FIGS. 5-7. PS 400can include each of the sensor pixel configurations shown in FIGS. 8-14(e.g., on a different sensor panels 121).

FIG. 8 shows an example first sensor pixel 501 a. First sensor pixel 501a can include a microlens 801, a color filter 802, a spacer layer 803,and a respective portion of a substrate 804. Substrate 804 can include aphotodiode 805. First sensor pixel 501 a can include one or morephotodiodes 805. Each of the one or more photodiodes 805 can be fastphotodiodes 807. First sensor pixel 501 a can be a G, B, R, and/or PDsensor pixel 501 r,g,b,pd. First sensor pixel 501 a can correspond tofast sensor pixels 503.

Microlens 801 can be dome-shaped. A dome-shaped microlens 801 caninclude a flat top and thus have a trapezoidal-hemispherical shape.Microlens 801 can be fully hemispherical (e.g., ½ of a sphere). As withany features disclosed herein, microlens 801 is optional and can beabsent.

Microlens 801 can be disposed in a different optical location (e.g.,optically downstream of color filter 802). Microlens 801 can represent asingle, unitary microlens 801 that covers photodiode 805. The lateralposition of microlens 801 with respect to photodiode 805 can vary acrosssensor panel 121.

Microlens 801 of centrally disposed sensor pixels 501 can coverphotodiode 805 by being disposed directly vertically above photodiode805. Microlens 801 of peripherally disposed sensor pixels 501 can coverphotodiode 805 by being laterally offset from photodiode 805. Microlens801, if present, can be made from a drop of clear resin. The resin canbe a polymer. The resin can be a glass.

Color filter 802 can be configured to admit a first spectrum of lightand block remaining light. For example, a blue color filter can admitblue light while blocking non-blue light. As addressed above, multipledifferent color filters 802 can form a color filter array.

Each of FIGS. 5-7 shows an example color filter array with repeatingcolor filter patterns. FIG. 7 shows a pure Bayer pattern. FIG. 5 shows aBayer with phase detection pattern. Color filter 802 can be green, blue,red, clear, infrared, black, violet, cyan, magenta, yellow, etc. In someembodiments, color filter 802 can lie optically downstream of microlens801 and optically upstream of photodiode 803. In other embodiments,color filter 802 can lie optically upstream of microlens 801 (notshown). In further embodiments, color filter 802 can be absent for someor all sensor pixels 501. Color filter 802, if present, can be made froma rectangular (e.g., square) resin. The resin can be a polymer. Theresin can be a glass.

Spacer layer 803 can represent an absence of material (e.g., a void).Spacer layer 803 can be absent in some embodiments. In otherembodiments, spacer layer 803 can be charged with a clear resin. Theresin can be a polymer. The resin can be a glass. Spacer layer 803, ifpresent, gives microlens 801 additional vertical room to converge lightonto photodiode 805. Although not shown, circuitry and other features ofsubstrate 804 (e.g., photodiode 805) can protrude into spacer layer 803according to various embodiments. In some embodiments, spacer layer 803can be optically downstream of microlens 801 and color filter 802. Inother embodiments, spacer layer 803 can be optically upstream ofphotodiode 805.

Substrate 804 can be made from silicon. Substrate 804 can includecircuitry (not shown). Substrate 804 can include multiple photodiodes805. Photodiode 805 can be a complementary metal-oxide semiconductor(“CMOS”) photosensor, a charge-coupled device (“CCD”) photosensor, orany other kind of photosensor. Photodiode 805 can be opticallydownstream of microlens 801 and color filter 802.

Photodiode 805 can have a light receiving surface area 805 a and aphotocharge well 805 b. Although shown as linear, light receivingsurface area 805 a can be non-linear (e.g., include peaks and valleys).Light receiving area 805 a can be the exposed portion of photodiode 805when viewed from a top plan perspective.

Photodiode 805 can be configured to receive scene light through lightreceiving area 805 a (also called an unmasked/unshielded photosensitivearea). Photodiode 805 can convert the received scene light intophotocurrent at a predetermined efficiency level. Photodiode 805 canstore the photocurrent as photocharge in photocharge well 805 b.Photocharge well 805 b can have a maximum capacity.

Photodiode 805 can produce a maximum photometric in response tophotocharge well 805 b being saturated (e.g., filled). Photodiode 805can produce the maximum photometric in response to photocharge well 805b being oversaturated. Oversaturation can occur when (a) morephotocharge accumulates in well 805 b than PS 400 is capable of readingand/or (b) photocharge deposited in well 805 b leaks due to well 805 bbeing saturated.

Photodiode 805 can produce a minimum photometric (e.g., a zero reading)in response to photocharge well 805 b being empty (e.g., beingcompletely empty or having a photocharge quantity less than apredetermined activation quantity). As discussed with reference to FIGS.12 and 13, photodiode 805 can represent multiple different photodiodes805.

Sensor panel 121 can be an aspect of PS 400 (e.g., a sensor 404). PS 400can integrate sensor pixels 501 on a global or rolling basis.Integration can include a series of operations. First, PS 400 can selectcertain sensor pixels 501 (e.g., a row of sensor pixels 501, all sensorpixels 501). Second, PS 400 can clear (also called resetting) theselected sensor pixels 501 (e.g., by emptying photocharge wells 805 b ofthe selected sensor pixels 501). Third, PS 400 can wait while theselected sensor pixels 501 accumulate photocurrent as photocharge.Fourth, PS 400 can read out the photocharge levels of each photodiode805 of the selected sensor pixels 501.

Integration time can be the time difference between readout andclearing. During rolling integration, different sensor pixels 501 can beintegrated at different absolute times but the integration time of eachof the sensor pixels 501 can be the same.

FIGS. 9 and 10 discuss a second sensor pixel 501 bb. FIG. 9 shows afirst embodiment thereof 501 bbx. FIG. 10 shows a second embodimentthereof 501 bby. Second sensor pixel 501 bb can include a microlens 801,a color filter 802, a respective portion of a substrate 804, aphotodiode 805, and shielding (also called masking) 811. Second sensorpixel 501 bb can be equal to (e.g., substantially the same as) firstsensor pixel 501 a (FIG. 8) except for shielding 811. Second sensorpixel 501 bb can be a G, B, R, or PD sensor pixel 501 r,g,b,pd. Secondsensor pixel 502 can include one or more slow photodiodes 805, 806.

Shielding 811 can be opaque (e.g., substantially light impermeable orcompletely light impermeable). In some embodiments, shielding 811 can bemetallic (e.g., aluminum). In other embodiments, shielding 811 can bemade from an opaque (e.g., dark black) resin. FIG. 9 shows possiblelocations of shielding 811 including a first location 812 and a secondlocation 813.

Referring to FIG. 9, shielding 811 can be present at either of locations812, 813. When at location 812, shielding 811 can be applied directlyabove photodiode 805. When at location 813, shielding 811 can be applieddirectly below color filter 802. Although not shown, shielding 811 canexist at any location (e.g., in the middle of spacer layer 803, abovemicrolens 801, between microlens 801 and color filter 802, and so on).

Shielding 811 can have a closed outer perimeter and a closed innerperimeter. Shielding 811 can define a central aperture 821. Shielding811 can enable passage of light through central aperture 821. Shielding811 can reduce the light receiving area 805 a of photodiode 805. Asshown in FIG. 10, light receiving area 805 a is now only located at thecenter of photodiode 805. The periphery of photodiode 805 is masked 805c. As shown in FIG. 10, color filter 802 can be disposed within centralaperture 821.

FIG. 11 shows an example third sensor pixel 501 c. In some embodiments,third sensor pixel 501 c can be identical to first sensor pixel 501 a(FIG. 8) and/or second sensor pixel 501 bb (FIG. 9 or FIG. 10), exceptfor the disposition of shielding 811.

In third sensor pixel 501 c, shielding 811 can be disposed over a first(e.g., right) portion (e.g., half) 814 x of photodiode 805 to definelight receiving area 805 ax. As shown in broken lines, shielding 811 canalternatively be disposed over a second (e.g., left) portion (e.g.,half) 814 y of photodiode 805 to define light receiving area 805 ay.Some third sensor pixels 501 c can include shielding at first location814 x while other third sensor pixels 501 c can include shielding atsecond location 814 y.

In some embodiments, two third sensor pixels 501 c can be disposeddirectly next to each other to define a phase detection pixel pair. Forexample, PD sensor pixels 501 pd in FIG. 5 can represent a phasedetection pixel pair. One of the PD sensor pixels 501 pd can includeshielding 811 at 814 x (see FIG. 11), while the other of the PD sensorpixels 501 pd can include shielding at 814 y (see the broken lines inFIG. 11 depicting the alternate embodiment).

FIG. 12 shows an example fourth sensor pixel 501 d including multiple(e.g., four) photodiodes 805. Each can be fast or slow photodiodes 806,807. Any of the sensor pixels 501 discussed herein can include anynumber of (e.g., one, two, four, etc.) of photodiodes 805. Second sensorpixels 501 bb can be fourth sensor pixels 501 d. PS 400 can beconfigured to independently read out each photodiode 805 of each of thesensor pixels 501 (e.g., fourth sensor pixel 501 d). Fourth sensor pixel501 d can include any of the features discussed with reference to FIGS.8-11 (e.g., a microlens 801, a color filter 802, a spacer layer 803,etc.).

Fourth sensor pixel 501 d can include peripheral shielding 811. Asshown, peripheral shielding 811 can be discontinuous due to small gaps890 between adjacent photodiodes 805. Although not shown, peripheralshielding 811 can be continuous.

Peripheral shielding 811 can define a central aperture 821, whichreveals light receiving area 805 a, which is shown to occupy apie-shaped corner of each photodiode. Shielding 811 can be absent orrearranged (discussed below). A microlens 801 can cover photodiodes 805of fourth sensor pixel 501 d (see FIGS. 8-11). An optical axis of lightconverged by microlens 801 can pass through the center of the effectivecircle produced by the sum of light receiving areas 805 a.

FIG. 13 shows an example fifth sensor pixel 501 e. Shielding 811 can bedisposed to define different central apertures 821. Each centralaperture 821 can be defined over the two-dimensional centroid of onephotodiode 805. Each central aperture 821 can have a different size.Each central aperture 821 can be circular, square, diamond shaped, etc.According to some embodiments, one photodiode 805, 1201 can lackshielding 811 and the remaining three photodiodes 805, 1202-1204 caninclude shielding 811 defining three respective central apertures 821 ofdifferent sizes.

FIG. 14 shows an example sixth sensor pixel 501 f including multiple(e.g., two) photodiodes 805. Sixth sensor pixel 501 f can includeelongated photodiodes 805. Adjacent sides of elongated photodiodes 805can be covered by shielding 811. The shielding 811 can include a firstshield 811, 1401 and a second shield 811, 1402. Therefore, oppositesides of elongated photodiodes 805 can define light receiving surfaces805 a such that shielding 1401, 1402 is disposed in-between the pair oflight receiving surfaces 805 a. A gap 1403 is shown as separatingphotodiodes 805. In some embodiments, gap 1403 can be absent such thatadjacent sides of photodiodes 805 are in direct contact with each other.In some embodiments (not shown), shielding 1401, 1402 can be swappedwith light receiving surfaces 805 a such that the pair of lightreceiving surfaces 805 a are disposed in-between the shielding 1401,1402.

As previously discussed, sensor pixels 501 can be slow sensor pixels 502or fast sensor pixels 503. Similarly, photodiodes 805 can be slowphotodiodes 806 or fast photodiodes 807. All else being equal, fastsensor pixel photodiodes 807 can saturate more quickly with photochargethan slow sensor pixel photodiodes 806. Slow sensor pixels 502 and slowphotodiodes 806 can have a first photosensitivity, a first lightreceiving area 805 a, and a first photocharge well depth 805 b. Fastsensor pixels 503 and fast photodiodes 807 can have a secondphotosensitivity, a second light receiving area 805 b, and a secondphotocharge well depth 805 b.

In some embodiments, the first photosensitivity can be less than thesecond photosensitivity, the first light receiving area 805 a can beequal to the second light receiving area 805 a, and the first well depth805 b can be equal to (e.g., the same or substantially the same as) thesecond well depth 805 b. In other embodiments, the firstphotosensitivity (which can be surface area independent) can be equal tothe second photosensitivity, the first light receiving area 805 a can besmaller than the second light receiving area 805 a, and the first welldepth 805 a can be equal to the second well depth 805 a. In furtherembodiments, the first and second photosensitivities can be equal, thefirst and second light receiving areas 805 a can be equal, and the firstwell depth 805 b can be deeper than the second well depth 805 b. Slowsensor pixels 502 and slow photodiodes 806 can include any combinationof the above saturation delaying features.

Saturation time can be the amount of time, all else being equal, that aparticular one of the sensor pixels takes to saturate with photochargein well 501 b. Fast sensor pixels 503/fast photodiodes 807 can beconfigured to saturate faster (i.e., have lower saturation times) thanslow sensor pixels 502/slow photodiodes 806. However, during use, if afast sensor pixel 503/fast photodiode 807 maps to a dark scene objectand a slow sensor pixel 502/slow photodiode 806 maps to a bright sceneobject, then the slow sensor pixel 502/slow photodiode 806 may saturatequicker than the fast sensor pixel 503/fast photodiode 807.

Unless otherwise indicated, the present disclosure discusses saturationtime in terms of configuration, which is independent of any specificimaging event. For convenience, the present disclosure uses thefollowing conventions to compare the saturation times of differentsensor pixels 501 and different photodiodes 805. These conventions areonly for convenience and ease of explanation and are not intended tolimit the claimed inventions.

First, each of the sensor pixels 501 has the same two-dimensional area,which is covered by a singlet-photodiode 805 (e.g., as depicted in FIGS.8-11), two dual-photodiodes 805 (e.g., as depicted in FIG. 14), or fourquad-photodiodes 805 (e.g., as depicted in FIGS. 12 and 14). As aresult, the light receiving area 805 a of any unshielded sensor pixel(e.g., first sensor pixel 501 a) is assumed to be constant, no matterwhether the sensor pixel (e.g., first sensor pixel 501 a) includes asinglet-photodiode 805, dual-photodiodes 805, or quad-photodiodes 805.

Second, the photosensitivity (defined to be area independent) of eachsinglet, dual, and quad-photodiode 805 is assumed to be the same. Third,the photocharge well capacity 805 b of a singlet photodiode is assumedto be twice as large as the well capacity 805 b of a singledual-photodiode, which is assumed to be twice as large as the wellcapacity 805 b of a single quad-photodiode. Fourth, no sensor pixels 501include a microlens 801. Fifth, the sensor pixels 501 are located in thesame general area of sensor panel 121 and receive the same light.

Given these assumptions: First sensor pixel 501 a (and its correspondingphotodiode(s) 805) can have the shortest saturation time (i.e., bestructurally configured to have the shortest saturation time) due to thelack of shielding 811. Third sensor pixel 501 c and sixth sensor pixel501 f (and their corresponding photodiode(s) 805) can have equal (e.g.,substantially similar) intermediate saturation times. Second sensorpixel 501 b and fourth sensor pixel 501 d (and their correspondingphotodiode(s) 805) can have the longest saturation times. All else beingequal, unshielded singlet, dual and quad-photodiodes 805 can have fastand equal saturation times.

Each photodiode 805 of fifth sensor pixel 501 e can have a differentsaturation time. Photodiode 805, 1201 with the least shielding 811 canhave the fastest saturation time while photodiode 805, 1202 can have theslowest saturation time.

Referring to FIG. 5, PD sensor pixels 501 pd can be third sensor pixels501 c. Each PD sensor pixel 501 pd can include only a single photodiode805 or multiple photodiodes 805. Adjacent PD sensor pixels 501 pd canhave opposite shielding 811. For example, the right PD sensor pixel 501pd of FIG. 5 can have shielding at first location 814 x while the leftPD slow sensor pixel 501 pd of FIG. 5 can have shielding at secondlocation 814 y. Adjacent PD sensor pixels 501 pd can form a phasedetection pixel group. A single microlens 801 can cover each phasedetection pixel group. Although not shown, sensor panel 121 of FIG. 5can include multiple phase detection pixel groups.

In FIG. 5, each PD sensor pixel 501 pd can be a slow sensor pixel 502.R, G, and B pixels 501 r,g,b can be fast sensor pixels 503. R, G, and Bsensor pixels 501 r,g,b can each be first sensor pixels 501 a. R, G, andB sensor pixels 501 r,g,b can each include one or more photodiodes 805.According to various embodiments: each photodiode 805 can be unshielded;some of R, G, and B sensor pixels 501 r,g,b can be unshielded whileother R, G, and B sensor pixels 501 r,g,b can be shielded; only one ofR, G, and B sensor pixels 501 r,g,b can include shielded sensor pixels(e.g., 1/10 of all B sensor pixels 501 b) can include shielding 811;none of G sensor pixels 501 g can include shielding 811; the R, G,and/or B sensor pixels 811 r,g,b with shielding 811 can be second sensorpixels 501 bb, etc.

Referring to FIG. 6, PD sensor pixels 501 pd can be slow sensor pixels502 (i.e., slower than other PD sensor pixels 501 pd such as fast sensorpixels 503—discussed below). PD sensor pixels 501 pd can be secondsensor pixels 501 bb (with a smaller central aperture 821), third sensorpixels 501 c, fourth sensor pixels 501 d (with a larger central aperture821), fifth sensor pixels 501 e, or sixth sensor pixels 501 e. R, G, andB sensor pixels 501 can each be fast sensor pixels 503 (i.e., fasterthan other PD sensor pixels 501 pd, such as slow sensor pixels 502). R,G, and B sensor pixels 501 can be first sensor pixels 501 a, secondsensor pixels 501 bb (with a larger central aperture 821), fourth sensorpixels 501 d (with a larger central aperture 821), or fifth sensorpixels 501 e.

Referring to FIG. 7, each of the sensor pixels 501 can be asecond/fourth/fifth sensor pixel 501 bb/501 d/501 e. If each of thesensor pixels 501 is a fifth sensor pixel 501 e, some sensor pixels 501can include smaller apertures 821 and others can include largerapertures 821. Each R, G, and B sensor pixel 501 r,g,b can includemultiple photodiodes 805, but a certain percentage of G sensor pixels501 g can include shielding as in fourth sensor pixel 501 d, while theremaining sensor pixels 501 can be unshielded (see, for example, firstsensor pixel 501 a). Each of the sensor pixels 501 can be a sixth sensorpixel 501 f. Some of the sixth sensor pixels 501 f can be rotated 90degrees with respect to the other sixth sensor pixels 501 f.

FIG. 15 shows an example method of automatic exposure control (AEC). Aswith all methods and operations disclosed herein, PS 400 can beconfigured to perform the method of FIG. 15.

At block 1502, PS 400 can capture a scene. PS 400 can do so byintegrating slow photodiodes 805 of first sensor pixels 501 for a firstamount of time and integrating fast photodiodes 805 of second sensorpixels 501 for a second amount of time. The slow photodiode 805 can beconfigured to saturate with photocharge slower than the fast photodiode805. As previously discussed, a slow photodiode 805 can be configured tosaturate with photocharge slower than a fast photodiode 805, but stillaccumulate a greater amount of photocharge than the fast photodiode 805due to differences in scene light intensity.

Although integrated for the same amount of time, the fast and slowphotodiodes 806 can be integrated at different absolute times (e.g., ifPS 400 performs rolling integration). The slow photodiode 805 can be anyslow photodiode 805 disclosed herein. The fast photodiode 805 can be anyfast photodiode 805 disclosed herein.

At block 1504, PS 400 can re-capture the scene. To do so, PS 400 canintegrate the fast photodiode 805 for a second amount of time to producea newer second photometric. The second amount of time can be based onthe original first photometric. The second amount of time can be lessthan the first amount of time.

Because sensor pixels 501 comprise photodiodes 805, any operationsperformed by a photodiode 805 can be understood as being performed byits corresponding sensor pixel 501. For example, assume that sensorpixel X comprises photodiodes Y and Z. When photodiode Y is integrated,sensor pixel X can be understood as being integrated. And whenphotodiode Z is integrated, sensor pixel X can be understood as beingintegrated. The same concept applies to other features (e.g., propertiesor characteristics) of photodiodes. For example, if photodiode Y has afirst photocharge capacity and photodiode Z has a second photochargecapacity, then sensor pixel X can be understood as including the firstphotocharge capacity and/or the second photocharge capacity.

Therefore, any reference in the present disclosure about a photodiode805 (including the properties thereof) can be replaced with a referenceto the sensor pixel 501 including the photodiode 805. And any discussionabout a sensor pixel 501 (including the properties thereof) can bereplaced with a reference to the one or more photodiodes 805corresponding to the sensor pixel 501.

FIG. 16 shows an example method of automatic exposure control (AEC). Aswith all methods and operations disclosed herein, PS 400 can beconfigured to perform the method of FIG. 16.

The method can generally include capturing a scene by integrating afirst sensor pixel 501 (e.g., a first photodiode 805 thereof) for afirst amount of time to produce an original first photometric andintegrating a second sensor pixel 501 (e.g., a second photodiode 805thereof) for the first amount of time to produce an original secondphotometric. The first sensor pixel 501 (e.g., the first photodiode 805thereof) can be configured to saturate with photocharge slower than thesecond sensor pixel 501 (e.g., the second photodiode 805 thereof).

The method can include re-capturing the scene by integrating the secondsensor pixel 501 (e.g., the second photodiode 805 thereof) for a secondamount of time to produce a newer second photometric. The second amountof time can be based on the original first photometric. The secondamount of time can be less than the first amount of time.

Referring to FIG. 16, and at block 1602, PS 400 can integrate aplurality of first sensor pixels 501 and a plurality of second sensorpixels 501. The integration can happen simultaneously or sequentially.The first and second sensor pixels 501 can be any of sensor pixels 501a-501 f. The first sensor pixels 501 can be slow sensor pixels 502 andthus include one or more slow first photodiodes 806. The second sensorpixels 501 can be fast sensor pixels 503 and thus include one or moresecond fast photodiodes 807.

The first sensor pixels 501 and the second sensor pixels 501 can bestructurally identical and each include a slow photodiode 805 and a fastphotodiode 805 (e.g., fifth sensor pixel 501 e of FIG. 13). The slowphotodiode 806 can be any photodiode of sensor pixels 501 a-501 f. Thefast photodiode 807 can be any photodiode of sensor pixels 501 a-501 f.The slow photodiode 806 can have a smaller light receiving area 805 athan the fast photodiode 807. The slow photodiode 806 can be otherwiseidentical (e.g., configured to be identical) as the fast photodiode 807.

At block 1602, PS 400 can read out first analog photometrics of the slowphotodiodes 806 and second analog photometrics of the fast photodiodes807. As discussed above, read-out can represent the final step inphotodiode integration.

At block 1604, PS 400 can create an image (e.g., a mosaic and/or afull-color image) with the photometrics. Block 1604 can includeconverting the first and second analog photometrics into first andsecond digital photometrics (e.g., channel values). When the presentdisclosure refers to photometrics, the photometrics can be in analog ordigital form. The photometrics can quantify light brightness (e.g.,intensity).

An image can include many different image pixels. In some embodiments,each image pixel can spatially map to one sensor pixel. For an example,an image pixel with coordinates (1, 1) can map to a sensor pixel withcoordinates (1, 1), an image pixel with coordinates (x, y) can map to asensor pixel with coordinate (x, y), and so on. In other embodiments,each image pixel can map to a cluster of multiple sensor pixels, or viceversa.

Each image pixel can have one or more color channels. A color channelcan be one of multiple predetermined options, determined according to adesired color space. In RGB color space, each image pixel can include ared, a green, and/or a blue color channel. Other color spaces (e.g.,CIE) are consistent with the present disclosure. As used herein the termcolor can also refer to non-visible light spectrum such as infraredlight.

Each color channel can have a color value falling within a predeterminedrange such as 0-255 (8-bits per channel), 0-511 (9-bits per channel),0-1023 (10-bits per channel), 0-2047 (11-bits per channel), and so on.The color value can indicate a magnitude of the color channel. At leastone color value of an image pixel can be based on the photometric(s)captured by the photodiode(s) mapping to the image pixel. For example, acolor value of zero can indicate a color channel with zero magnitude.For clarity, image pixels are considered to include a color channel whenthe color value thereof is zero. Image pixels do not include a colorchannel when the color value thereof is unassigned.

Images can have different stages and exist in different states. Stagescan include a mosaic stage and a full-color stage. States can include atransient state and a stable state. When the present disclosure refersto an image, the image can be a portion or patch of a larger image. Insome embodiments, the image can represent an entire and complete image.In some embodiments, image can be a video frame.

In a mosaic stage, each image pixel can include fewer color channelsthan are required for a full-color image (as determined by the desiredcolor space). According to some examples, each image pixel in an imagemosaic (i.e., an image in a mosaic stage) can include a single colorchannel, and thus a single color value. Each image pixel in an imagemosaic can include a single color channel but a different color valuefor each photodiode of the corresponding sensor pixel. In a full-colorstage, each image pixel includes a number of color channels equal to thenumber required by the desired color space. For example, in RGB colorspace, each image pixel includes three color channels (red, green, andblue).

A mosaic can have a color pattern. PS 400 can initially capture an imagein a mosaic stage with a color pattern matching the arrangement of thecolor filters. For example, if PS 400 captured an image mosaic withsensor panel 121 of FIG. 7, the top left image pixel would have a bluecolor channel and color value(s) determined by photocharge(s) collectedby the photodiode(s) of the corresponding sensor pixel. Therefore, insome examples, each of the sensor pixels 501 shown in FIGS. 5-7 can mapto different image sub-pixels (not shown) in the first mosaic. Eachsub-pixel can map to one photodiode.

PS 400 can convert a mosaic (e.g., a first mosaic) into a full-colorimage via demosaicing. Demosaicing can include full-color interpolation.Full-color interpolation can include assigning multiple color channels(e.g., two, three, four, five, etc.) to each image pixel. For example,if PS 400 observes RGB color space, each image pixel can be assigned ared color channel, a green color channel, and a blue color channelduring full-color interpolation.

In some embodiments, demosaicing can include remosaicing. For example,PS 400 can remosaic the first mosaic into a second mosaic, then performfull-color interpolation on the second mosaic. Remosaicing can includebinning. For example, each image pixel in the first mosaic can havemultiple color values (e.g., one per photodiode spatially mapping to theimage pixel) for the same color channel. During binning, the colorchannels for each image pixel can be combined (e.g., averaged).

Any stage of an image can be in a transient state, where the imageexists as signals in PS 400 or a stable state, when the image ispreserved in memory (e.g., volatile memory and/or non-volatile memory).Whether in transient form or stable form, an image can have aresolution, which quantifies the detail that the image holds. Thesmallest unit of resolution can be an image pixel.

Stable image and transient images can have image pixels stored incompressed form. For example, a JPEG image is a full-color image. Thethree color channels of each image pixel in a JPEG image are stored in acompressed format (e.g., in the frequency domain). Upon accessing a JPEGimage, PS 400 can use a codec to unpack the three color channels of eachimage pixel.

Block 1604 can include demosaicing and thus result in a full-colorimage. Demosaicing is discussed above. PS 400 can decline to prepare(e.g., produce, create) a full-color image until AEC is complete.

At block 1606, PS 400 can determine (e.g., estimate) scene dynamicrange. Estimating scene dynamic range can include using slow photodiodes806 to find a ceiling of the scene dynamic range and fast photodiodes807 to find a floor of the scene dynamic range. Block 1606 can includeblocks 1606 a-c. During block 1606, PS 400 can build one or more ofhistograms 1710, 1720, 1730, 1740 (discussed below).

At block 1606 a, PS 400 can normalize the first photometrics withrespect to the fast photodiodes 807. As addressed above, the slowphotodiodes 806 can have a different configuration than the fastphotodiodes 807 to result in different saturation times. Therefore, thefirst photometrics can lie on a first scale corresponding to the slowphotodiodes 806 and the second photometrics can lie on a second scalecorresponding to the fast photodiodes 807. The first and second scalescan be different.

PS 400 can normalize (e.g., scale, compensate) the first photometricswith respect to the fast photodiodes 807 to put the first photometricson the same scale as the second photometrics. Therefore, in someembodiments, the second photometrics are not normalized. For example, afirst photometric of X can indicate the same scene intensity (e.g.,brightness) as a second photometric of 2*X. Therefore, PS 400 cannormalize the first photometrics by increasing (e.g., upscaling) thefirst photometrics. According to the above example, PS 400 couldmultiply each first photometric by two. PS 400 can normalize based on adifference between (e.g., ratio between) light receptive area 805 a ofthe slow photodiodes 806 and light receptive area 805 a of the fastphotodiodes 807.

At block 1606 b, PS 400 can find (e.g., determine, estimate) a scenedynamic range ceiling. At block 1606 b, PS 400 can attempt to constructscene dynamic range top end 1713 of histogram 1710 (discussed below).

PS 400 can determine whether a sufficient number (e.g., a sufficientpercentage) of normalized (e.g., scaled, compensated) first photometricslie above a saturation photometric of the fast photodiodes. Thesaturation photometric can be a fixed and/or predetermined photocharge.PS 400 can make this determination by building histograms 1720 and 1730.

If an insufficient number of first photometrics lie above the saturationphotometric (of the fast photodiodes 807), then PS 400 can assess thatno oversaturation clipping of the second photometrics has occurred. If asufficient number of normalized first photometrics lie above thesaturation photometric, then PS 400 can assess that oversaturationclipping of the second photometrics has occurred.

At block 1606 c, PS 400 can find (e.g., estimate) a scene dynamic rangefloor. At block 1606 c, PS 400 can attempt to construct scene dynamicrange bottom end 1711 of histogram 1710 (discussed below). PS 400 canuse the second photometrics, but not the first photometrics to find thedynamic range floor (PS 400 can similarly exclude the thirdphotometrics, discussed below, finding scene dynamic range ceilingduring block 1606 b).

At block 1608, PS 400 can generate (e.g., determine, calculate) one ormore metrics based on the outcomes of block 1606. PS 400 can determinewhether to perform automatic exposure control (AEC) based on the one ormore metrics. During block 1608, PS 400 can produce one or morehistograms, as discussed below with reference to FIG. 17. Block 1608 caninclude blocks 1608 a-1608 d.

PS 400 can perform block 1608 a in response to finding oversaturationclipping at block 1606 b and/or undersaturation clipping at block 1606c. PS 400 can perform block 1606 (e.g., blocks 1606 a-1606 c) duringblock 1608 a.

At block 1608 a, PS 400 can quantify the number of oversaturated fastphotodiodes 807 and the number of undersaturated fast photodiodes 807.PS 400 can do so by analyzing histogram 1720 in light of histograms 1710and/or 1730. Therefore, PS 400 can use the histograms discussed withreference to FIG. 17 to quantify oversaturated and/or undersaturatedphotodiodes 805.

Given the normalized first photometrics, PS 400 can estimate how manyfast photodiodes 807 should be fully saturated. PS 400 can assume thatthe remaining fast photodiodes 807 producing a maximum photometric areoversaturated. PS 400 can perform the reverse operation to estimate howmany fast photodiodes 807 are undersaturated based on the normalizedthird photometrics.

At block 1608 b, PS 400 can quantify a magnitude of oversaturation and amagnitude of undersaturation. PS 400 can individually quantify (e.g.,estimate) the magnitude of each oversaturation and the magnitude of eachundersaturation. Alternatively or in addition, PS 400 can find a medianor average of the normalized first photometrics exceeding theoversaturation threshold of the fast photodiodes 807. PS 400 can comparethe median or average with the saturation capacity (i.e., theoversaturation threshold) of the fast photodiodes 807.

For example, if the oversaturation threshold is 50 photocharge units andthe normalized first photometrics includes 48 photocharge units, 52photocharge units, and 54 photocharge units, PS 400 can compare themedian or average of 53 photocharge units (48 photocharge units can beignored for being under the oversaturation threshold) with theoversaturation threshold of 50 photocharge units. PS 400 can perform thereverse with respect to the undersaturation threshold of the fastphotodiodes 807 (which can be zero or one).

At block 1608 c, PS 400 can perform statistical measurements on thespread of second photometrics to determine whether the secondphotometrics are clustered or spread out. PS 400 can assign a dispersalrating to the second photometrics based on the statistical measurements.Dispersal can be judged by analyzing whether histogram 1720 includesclusters of photometrics.

At block 1608 d, PS 400 can determine whether to execute AEC based onone or more factors. Factor one can be the number of oversaturated fastphotodiodes 807. Factor two can be the magnitude of oversaturation forthe oversaturated fast photodiodes 807 (as approximated with thecomparison between the median or average of the first photometrics withthe oversaturation capacity of the fast photodiodes 807). Factor threecan be the number of undersaturated fast photodiodes 807. Factor fourcan the magnitude of undersaturation for the undersaturated fastphotodiodes 807 (as approximated with the comparison between the medianor average of the third photometrics with the undersaturationthreshold/capacity of the fast photodiodes 807). Factor five can be thedispersal rating. A high dispersal (i.e., photometrics being spread out)can weigh against an AEC adjustment and a low dispersal (i.e., secondphotometrics being clustered together) can weigh in favor of an AECadjustment. An example of a dispersal rating can be based on a standarddeviation of photometric density. For example, if the standard deviationis high, then dispersal can be low and if the standard deviation is low,then the dispersal can be high.

If PS 400 determines that AEC is unnecessary at block 1608 d, then PS400 can return to block 1602. If PS 400 determines that AEC is necessaryat block 1608 d, then PS 400 can proceed to block 1610. At block 1610,PS 400 can execute AEC. Techniques for performing AEC are discussedabove and can include adjusting photodiode 805 integration time,photodiode 805 gain, and/or aperture size.

As discussed, PS 400 can reduce integration time to reduce exposure. PS400 can increase integration time to increase exposure. The magnitude ofAEC can be based on any of the metrics of block 1608. For example,integration time can be reduced based on (e.g., at least based on) thenumber of oversaturated fast photodiodes 807 and the median/averageoversaturation magnitude thereof.

As shown schematically in FIG. 18, sensor pixels 501 can each include atleast three photodiodes 805 with different saturation times, e.g., firstslow photodiodes 806, second fast photodiodes 807, and third fastestphotodiodes 808 According to these embodiments, PS 400 can use thirdphotometrics from the fastest photodiodes 805 to find scene dynamicrange floor.

The terms slow, fast, and fastest are relative. The second fastphotodiodes 807 can be slower than the third fastest photodiodes 808.Put differently, the second fast photodiodes 807 can be thought of asintermediate-speed. The term fastest is also relative and conveys thatthe third photodiodes 808 are faster than both the first 806 and second807 photodiodes. The term fastest is not intended to convey that thethird photodiodes 808 are the fastest photodiodes 805 in all of sensorpanel 121 (although third photodiodes 808 can be in some embodiments).

Therefore at block 1606 a, PS 400 can normalize the third photometricsfrom the fastest photodiodes 808 to the fast photodiodes 807 (e.g., bydownscaling the third photometrics). And at block 1606 c, PS 400 canperform the opposite of block 1606 b where PS 400 uses the normalizedthird photometrics to assess whether undersaturation clipping of thesecond photometrics has occurred.

If an insufficient number of normalized third photometrics lie below aundersaturation photometric (the undersaturation photometric cancorrespond to the minimum amount of photocharge necessary to activatethe fast photodiodes 807) of the fast photodiodes 807, then PS 400 canassess that undersaturation clipping of the second photometrics has notoccurred. If a sufficient number of normalized third photometrics liebelow a undersaturation photometric of the fast photodiodes, then PS 400can assess that undersaturation clipping of the second photometrics hasoccurred.

As with all features disclosed herein, the third photometrics areoptional. Various embodiments can lack fastest photodiodes and thus theability to produce third photometrics.

FIG. 17 illustrates an example application of the methods of FIGS. 15and 16 through histograms including a scene dynamic range (DR) histogram1710, an original fast photodiode DR histogram 1720, an original slowphotodiode DR histogram 1730, and a new fast photodiode histogram 1740.Each histogram can have a different X-axis scale.

Photometric value (e.g., magnitude) increases along each X axis.Therefore undersaturated (e.g., zero) photometrics are on the left sidehistogram and oversaturated (e.g., maximum capacity) photometrics are onthe right side. Photometric frequency runs along each Y axis. Therefore,rare photometric magnitudes are short (or nonexistent) while abundant(e.g., frequent) photometric magnitudes are tall.

Referring to histogram 1710, the scene dynamic range includes a lowbrightness (e.g., intensity) portion 1711 corresponding to the bottom ofthe scene dynamic range, a medium brightness portion 1712 correspondingto the middle of the scene dynamic range, and a high brightness portion1713 corresponding to a top of the scene dynamic range. Histograms 1720,1730, and 1740 follow the same hatching conventions as scene DRhistogram 1710.

Scene DR histogram 1710 can represent the true scene dynamic rangeand/or an approximate of the scene dynamic range. PS 400 can createscene DR histogram 1710 by building original fast photodiode histogram1720 and one or both of original slow photodiode histogram 1730 and anoriginal fastest photodiode histogram (not shown).

Referring to original fast photodiode histogram 1720, camera 101 hasfully captured the low end of the scene dynamic range with low magnitudesecond photometrics 1721 and fully captured the middle of the scenedynamic range with intermediate magnitude second photometrics 1722.Original fast photodiode histogram 1720 includes a large quantity offully saturated (i.e., maximum) magnitude photometrics 1723. Asindicated by the matching hatch patterns, fully saturated photometrics1723 correspond to scene dynamic range top 1713.

As shown in scene DR histogram 1710, scene dynamic range top can includemany different light intensities and thus result in multiple differentsecond photometrics. However, due to oversaturation, camera 101 hasassigned the same maximum intensity to the entire scene dynamic rangetop 1713. Fast photodiodes 807 have therefore clipped at least a portionof scene dynamic range top 1713.

PS 400 can infer that oversaturation clipping has occurred based on theunusual number (e.g., more than a predetermined number) of photometricswith the same maximum value. However, PS 400 may not have completeconfidence based on the original fast photodiode DR histogram 1720 alonebecause some or all of fully saturated photometrics 1723 can be correct(i.e., not clipped). Furthermore, PS 400 may be unable to determine themagnitude of clipping based on original fast photodiode DR histogram1720 alone.

Original slow photodiode DR histogram 1730 has fully captured scenedynamic range middle 1712 with intermediate first photometrics 1732 andscene dynamic range top 1713 with high first photometrics 1733.Histogram 1730 includes an unusual number (e.g., less than apredetermined number) of low (e.g., zero) magnitude readings 1731corresponding to scene dynamic range bottom 1711. Therefore, the slowphotodiodes 806 have undersaturation clipped 1731 x scene dynamic rangebottom 1711.

During the methods of FIGS. 15 and/or 16, PS 400 can normalize (e.g.,scale) original slow photodiode histogram 1730 and original fastestphotodiode histogram to original fast photodiode histogram 1720 (e.g.,block 1606 a, which can occur during block 1608 a). If the normalizedvalues exceed the ceiling of the second photometrics, then PS 400 canjudge that oversaturation clipping has occurred (e.g., block 1606 b,which can occur during block 1608 a). If the normalized values fallbelow the floor of the second photometrics, then PS 400 can judge thatundersaturation clipping has occurred (e.g., block 1606 c, which canoccur during block 1608 a).

PS 400 can estimate (e.g., determine) how many instances of secondphotometric oversaturation clipping occurred, and slide those instancesalong the X axis of original fast photodiode histogram 1720 past theceiling of histogram 1720 (e.g., block 1608 b). To do so, PS 400 canestimate a true X axis value for each of the ceiling (e.g., maximum)second photometrics by interpolating normalized first photometricsproduced by neighboring slow photodiodes 806. If the true X axis valueof a particular second photometric is greater than its current value, PS400 can slide the second photometric to the true X axis coordinate. PS400 can then supplement the histogram 1720 with each of the normalizedfirst photometrics.

Similarly, PS 400 can estimate (e.g., determine) how many instances ofsecond photometric undersaturation clipping occurred, and slide thoseinstances along the X axis of original fast photodiode histogram 1720past the floor of histogram 1720 (e.g., block 1608 b). To do so, PS 400can estimate a true X axis value for each of the floor (e.g., minimum)second photometrics by interpolating normalized first photometricsproduced by neighboring slow photodiodes 806. If the true X axis valueof a particular second photometric is greater than its current value, PS400 can slide the second photometric to the true X axis coordinate. PS400 can then supplement the histogram 1720 with each of the normalizedthird photometrics.

After approximating the complete (e.g., expanded) scene DR (e.g.,building histogram 1710), PS 400 can perform block 1608 d based onhistogram 1710. Put differently, PS 400 can determine whether to executeAEC, in what direction, and to what extent, based on histogram 1710. PS400 can execute AEC to minimize clipping of the scene dynamic range.

In some embodiments, clipping can be minimized by minimizing (i.e.,reducing) the aggregate number of clipping instances. In otherembodiments, clipping can be minimized by minimizing (i.e., reducing) aclipping index. PS 400 can assign a weight or magnitude to each instanceof second photometric clipping based on a difference between the true Xaxis value of the second photometric and the captured value. PS 400 cansum a magnitude of each weighted clipping instance to build a clippingindex. PS 400 can then perform AEC to reduce the clipping index.

Before creating new fast photodiode DR histogram 1740, PS 400 performedAEC by reducing integration time of sensor pixels 501. As shown in newfast photodiode DR histogram 1740, the fast photodiodes 807 bettercaptured the entire scene dynamic range (i.e., AEC has minimizedclipping). Because the scene dynamic range was wide, camera 101 wasunable to simultaneously prevent oversaturation clipping andundersaturation clipping. Therefore, a small portion of the scenedynamic range bottom was clipped (as evidenced by the large number ofminimum or undersaturated photometrics at the left of histogram 1740).And a small portion of the scene dynamic range top was clipped (asevidenced by the large number of maximum or saturated photometrics atthe right of histogram 1740).

Referring to FIG. 1, mobile device 100 can be a smartphone 100 a, atablet, a digital camera, or a laptop. Mobile device 100 can be adedicated camera assembly 100 b. Mobile device 100 can be mounted to alarger structure (e.g., a vehicle or a house). Mobile device 100 (or anyother device, such as a vehicle or desktop computer) can includeprocessing system 400.

As schematically shown in FIG. 4, mobile device 100 (or any otherdevice, such as a vehicle or desktop computer) can include a PS 400. PS400 can include one or more processors 401, memory 402, one or moreinput/output devices 403, one or more sensors 404, one or more userinterfaces 405, one or more motors/actuators 406, and one or more databuses 407.

Processors 401 can include one or more distinct processors, each havingone or more cores. Each of the distinct processors can have the same ordifferent structure. Processors 401 can include one or more centralprocessing units (CPUs), one or more graphics processing units (GPUs),circuitry (e.g., application specific integrated circuits (ASICs)),digital signal processors (DSPs), and the like. Processors 401 can bemounted on a common substrate or to different substrates. Processors 401can include read-out circuitry of sensor panel 121. Processors 401 caninclude circuitry defining an image processing pipeline.

Processors 401 are configured to perform a certain function, method, oroperation at least when one of the one or more of the distinctprocessors is capable of executing code, stored on memory 402 embodyingthe function, method, or operation. Processors 401 can be configured toperform any and all functions, methods, and operations disclosed herein.

For example, when the present disclosure states that PS 400 can performtask “X”, such a statement should be understood to disclose that PS 400can be configured to perform task “X”. Mobile device 100 and PS 400 areconfigured to perform a function, method, or operation at least whenprocessors 401 are configured to do the same.

Memory 402 can include volatile memory, non-volatile memory, and anyother medium capable of storing data. Each of the volatile memory,non-volatile memory, and any other type of memory can include multipledifferent memory devices, located at multiple distinct locations andeach having a different structure.

Examples of memory 402 include a non-transitory computer-readable mediasuch as RAM, ROM, flash memory, EEPROM, any kind of optical storage disksuch as a DVD, a Blu-Ray® disc, magnetic storage, holographic storage,an HDD, an SSD, any medium that can be used to store program code in theform of instructions or data structures, and the like. Any and all ofthe methods, functions, and operations described in the presentapplication can be fully embodied in the form of tangible and/ornon-transitory machine-readable code saved in memory 402.

Input-output devices 403 can include any component for trafficking datasuch as ports and telematics. Input-output devices 403 can enable wiredcommunication via Universal Serial Bus (USB®), DisplayPort®,High-Definition Multimedia Interface (HDMI®), Ethernet, and the like.Input-output devices 403 can enable electronic, optical, magnetic, andholographic, communication with suitable memory 403. Input-outputdevices can enable wireless communication via any wireless standard(e.g., a wireless local area network, a cellular connection, anear-field communication, and so on).

Sensors 404 can capture physical measurements of environment and reportthe same to processors 401. Sensors 404 can include photodiodes 805.

User interface 405 can enable user interaction with imaging system 40.User interface 405 can include displays (e.g., LED touchscreens (e.g.,OLED touchscreens), physical buttons, speakers, microphones, keyboards,and the like. User interface 405 can include display 42 and hard button43.

Motors/actuators 406 can enable processor 401 to control mechanicalforces. If camera 101 includes auto-focus, motors/actuators 406 can movea lens along its optical axis to provide auto-focus and/or performoptical image stabilization.

Data bus 407 can traffic data between the components of PS 400. Data bus407 can include conductive paths printed on, or otherwise applied to, asubstrate (e.g., conductive paths on a logic board), Serial ATAttachment (SATA) cables, coaxial cables, USB® cables, Ethernet cables,copper wires, and the like. According to some embodiments, data bus 407can include one or more wireless communication pathways and thus PS 400can be distributed across a network such as the internet. Data bus 407can include a series of different wires 407 (e.g., USB® cables) throughwhich different components of PS 400 are connected.

1. A method, comprising: capturing a scene by integrating a first sensorpixel for a first amount of time to produce an original firstphotometric and integrating a second sensor pixel for the first amountof time to produce an original second photometric, the first sensorpixel being configured to saturate with photocharge slower than thesecond sensor pixel; normalizing the original first photometric withrespect to the second sensor pixel; and re-capturing the scene byintegrating the second sensor pixel for a second amount of time toproduce a newer second photometric, the second amount of time beingbased on the normalized original first photometric, and the secondamount of time being less than the first amount of time.
 2. The methodof claim 1, further comprising: prior to re-capturing the scene,determining whether the original second photometric has been clippedbased on the normalized original first photometric.
 3. The method ofclaim 2, wherein determining whether the original second photometric hasbeen clipped based on the normalized original first photometriccomprises determining whether the normalized original first photometricwould oversaturate the second sensor pixel.
 4. The method of claim 3,further comprising: comparing the normalized original first photometricand a saturation capacity of the second sensor pixel; and setting thesecond amount of time based on the comparison.
 5. The method of claim 1,wherein the first sensor pixel and the second sensor pixel areconfigured to have equal photocharge capacity, the first sensor pixelcomprises shielding, and the first sensor pixel is configured to moreslowly saturate with photocharge than the second sensor pixel by virtueof the shielding.
 6. The method of claim 5, wherein the shielding isapplied directly over a photodiode of the first sensor pixel.
 7. Themethod of claim 6, wherein the first sensor pixel comprises a pluralityof photodiodes and the method further comprises: performing phasedetection autofocus based on the first sensor pixel.
 8. The method ofclaim 1, further comprising: prior to re-capturing the scene,determining whether the original second photometric is greater than orequal to a maximum photocharge capacity of the second sensor pixel;based on an affirmative determination, determining whether the originalsecond photometric has been clipped based on the original firstphotometric.
 9. A method comprising: capturing a scene with a scenedynamic range by integrating a plurality of first sensor pixels for afirst amount of time to produce a plurality of original firstphotometrics and integrating a plurality of second sensor pixels for thefirst amount of time to produce a plurality of original secondphotometrics, each of the plurality of first sensor pixels beingconfigured to saturate with photocharge slower than each of theplurality of second sensor pixels; normalizing the plurality of originalfirst photometrics with respect to the plurality of second sensorpixels; determining whether at least some of the plurality of secondsensor pixels clipped a top end of the scene dynamic range based on thenormalized plurality of original first photometrics; and based ondetermining that at least some of the plurality of second sensor pixelsclipped the top end of the scene dynamic range, re-capturing the sceneby integrating the plurality of second sensor pixels for a second amountof time to produce a plurality of newer second photometrics, the secondamount of time being less than the first amount of time.
 10. The methodof claim 9, comprising setting the second amount of time based on thefirst amount of time, the plurality of original first photometrics, andthe plurality of original second photometrics.
 11. A processing systemcomprising one or more processors configured to: capture a scene byintegrating a first sensor pixel for a first amount of time to producean original first photometric and by integrating a second sensor pixelfor the first amount of time to produce an original second photometric,the first sensor pixel being configured to saturate with photochargeslower than the second sensor pixel; normalize the original firstphotometric with respect to the second sensor pixel; and re-capture thescene by integrating the second sensor pixel for a second amount of timeto produce a newer second photometric, the second amount of time beingbased on the normalized original first photometric, and the secondamount of time being less than the first amount of time.
 12. The systemof claim 11, wherein the one or more processors are configured to:determine whether the original second photometric has been clipped basedon the normalized original first photometric prior to recapturing thescene.
 13. The system of claim 12, wherein the one or more processorsare configured to: determine whether the original second photometric hasbeen clipped based on the original first photometric by determiningwhether the normalized original first photometric would oversaturate thesecond sensor pixel.
 14. The system of claim 13, wherein the one or moreprocessors are configured to: compare the normalized original firstphotometric and a saturation capacity of the second sensor pixel; andset the second amount of time based on the comparison.
 15. The system ofclaim 11, wherein the first sensor pixel and the second sensor pixel areconfigured to have equal photocharge capacity and the first sensor pixelis configured to more slowly saturate with photocharge than the secondsensor pixel by virtue of shielding applied in the first sensor pixel.16. The system of claim 15, wherein the shielding is applied directlyover a portion of a photodiode of the first sensor pixel.
 17. The systemof claim 16, wherein the first sensor pixel comprises a plurality ofphotodiodes and the one or more processors are configured to: performphase detection autofocus based on the first sensor pixel.
 18. Thesystem of claim 11, wherein the one or more processors are configuredto: prior to re-capturing the scene, determine whether the originalsecond photometric is greater than or equal to a maximum photochargecapacity of the second sensor pixel; based on an affirmativedetermination, determine whether the original second photometric hasbeen clipped based on the original first photometric.
 19. The system ofclaim 11, wherein the one or more processors are configured to computethe second amount of time based on a difference between the originalfirst photometric, after being scaled, and a dynamic range ceiling. 20.The system of claim 11, wherein the one or more processors areconfigured to compute the second amount of time based on the originalfirst photometric after being normalized to a scale of the originalsecond photometric, the first sensor pixel having a different structurethan the second sensor pixel, the first sensor pixel and the secondsensor pixel both having the same kind of color filter